CrashReporter is a debugging facility in Mac OS X that logs information about all programs that crash. This technote describes CrashReporter in detail. It includes a description of the crash logs generated by CrashReporter, and how you can use these logs to debug your program. This technote is useful for anyone who develops Mac OS X user space software.

Introduction Anatomy of a Crash Log Basic Information Process Information Exception Information Backtrace Information Thread State Libraries Debugging Without Symbols Setting Up Your Build System Applications Libraries CrashReporter Limitations Further Reading Document Revision History

Introduction

Mac OS X's CrashReporter is a useful facility for learning about problems your application is experiencing in the field. CrashReporter performs two useful actions:

• When a program crashes, CrashReporter will record a crash log (typically into ~/Library/Logs/CrashReporter/<ProgramName>.crash.log), and inform the user by logging a message to the system log (/var/log/system.log).

• In addition, if the program that crashed is a GUI application, CrashReporter will present the user with a dialog asking them whether they want to submit a bug report to Apple (see Figure 1). If the user clicks Submit Report, CrashReporter displays another dialog showing the details of the report to be submitted (see Figure 2).

Figure 1: First CrashReporter dialog

Figure 2: Second CrashReporter dialog

In this technote I explain how to interpret crash logs that you have obtained from end users. In the first section I explain each part of the crash log in detail. Following that I show you how to get useful information from a crash log even if your program ships without debugging symbols. Finally, I explain some limitations of the current implementation.

Back to Top 

Anatomy of a Crash Log

A crash log has a number of different parts; in the following sections I describe each part in detail.

Basic Information

The first part of a crash log contains some basic system information. You can see an example in Listing 1.

Listing 1: Basic information

Host Name:      guy-smiley.local
Date/Time:      2004-08-31 09:13:18 +0100
OS Version:     10.3.4 (Build 7H63)
Report Version: 2

The most important piece of information here is the OS version. You should pay particular attention to the build number; each specific version of Mac OS X can have multiple variants distinguished only by their build numbers (this typically happens when Apple releases new hardware). In addition, make sure to look at the time and date to see if there's any suspicious patterns: if you get lots of crash logs that all occur at 12:00, you probably need to investigate your time handling code.

Back to Top 

Process Information

The next part of the crash log contains information about the process that crashed, as illustrated in Listing 2.

Listing 2: Process information

Command: TextEdit
Path:    /Applications/TextEdit.app/Contents/MacOS/TextEdit
Version: 1.3 (202)
PID:     1437
Thread:  0

The most important thing to note here is the name of the process that crashed. In some cases the actual process that died is not what you think. For example, if your application uses a helper tool to do some work, and that helper tool dies, you want to focus on the helper tool's code and not waste time debugging the application code.

Back to Top 

Exception Information

The third part of a crash log shows information about the processor exception that was the immediate cause of the crash. Listing 3 shows a typical example.

Listing 3: Exception information

Exception:  EXC_BAD_ACCESS (0x0001)
Codes:      KERN_PROTECTION_FAILURE (0x0002) at 0x00000000

The most common forms of exception are:

• EXC_BAD_ACCESS/KERN_INVALID_ADDRESS ó This is caused by the thread accessing unmapped memory. It may be triggered by either a data access or an instruction fetch; I describe how to tell the difference in a later section.

• EXC_BAD_ACCESS/KERN_PROTECTION_FAILURE ó This is caused by the thread trying to write to read-only memory. This is always caused by a data access.

• EXC_BAD_INSTRUCTION ó This is caused by the thread executing an illegal instruction.

In all cases the exception part of the crash log contains the address that triggered the exception (the exception address). In Listing 3 that address is 0x00000000.

Back to Top 

Backtrace Information

The fourth part of the crash log, which displays a backtrace for all of the threads in the crashed process, is typically the most interesting. Listing 4 shows a typical example.

Listing 4: Backtrace information

Thread 0 Crashed:
0   <<00000000>>   0x00000000 0 + 0
1   com.apple.CoreFoundation   0x90191790 __CFRunLoopRun + 0x350
2   com.apple.CoreFoundation   0x90195f1c CFRunLoopRunSpecific + 0x148
3   com.apple.HIToolbox        0x927d62d8 RunCurrentEventLoopInMode + Ö
4   com.apple.HIToolbox        0x927dca40 ReceiveNextEventCommon + 0x1Ö
5   com.apple.HIToolbox        0x927feb18 BlockUntilNextEventMatchingLÖ
6   com.apple.AppKit           0x92dd2a34 _DPSNextEvent + 0x180
7   com.apple.AppKit           0x92de93b0 -[NSApplication nextEventMatÖ
8   com.apple.AppKit           0x92dfd718 -[NSApplication run] + 0x21c
9   com.apple.AppKit           0x92eb9b80 NSApplicationMain + 0x1d0
10  com.apple.TextEdit         0x00007d98 0x1000 + 0x6d98
11  com.apple.TextEdit         0x00007c0c 0x1000 + 0x6c0c

In this example there is only one thread, so there's only one backtrace. In a multi-threaded process, there is one backtrace per thread. Thus, it's critical that you identify the thread that crashed. CrashReporter makes this easy by tagging that backtrace with the text "Thread <ThreadNumber> Crashed:". However, it's easy to overlook this text and erroneously assume that the Thread 0 is the one that crashed.

Each line the backtrace describes a nested function invocation (a frame), with the most recently executed function at the top and the least recently executed at the bottom. For each frame, the columns in the backtrace are as follows.

• The first column is the frame number, starting at 0 (indicating the function that crashed) and incrementing for each nested function call.

• The second column is the name of the library containing the code executing in this frame; this is derived by cross referencing the program counter address (from the next column) with the list of loaded libraries.

• The third column is the program counter address within the frame. For frame 0 this is typically the address of the instruction that caused the exception. For higher frames this is the return address for that frame. That is, for frame N it points to the next instruction that will execute when the function referenced by frame N - 1 returns.

• The fourth column is the symbolic name for the program counter address given in the third column. If you strip debugging symbols before shipping your application to end users, this column will just contain a hex number. You can work out the corresponding symbolic name using the technique described later in this document.

Finally, if your program is multi-threaded, you can often identify which thread is which by looking at the symbolic names deep within the backtrace. For example, in Listing 4, frame 9 lists NSApplicationMain as its symbolic address, indicating that this thread is the main thread. In contrast, the deepest frame for a pthread is always the routine _pthread_body.

Back to Top 

Thread State

The next part of the crash log contains a dump of the processor state of the thread that crashed. Listing 5 shows an example of this for PowerPC.

Listing 5: PowerPC thread state

PPC Thread State:
  srr0: 0x00000000 srr1: 0x4000d030                vrsave: 0x00000000
    cr: 0x44022482  xer: 0x20000004   lr: 0x90007018  ctr: 0x900074c0
    r0: 0xffffffe1   r1: 0xbfffeec0   r2: 0x00001003   r3: 0x10004005
    r4: 0x03000006   r5: 0x00000000   r6: 0x00000450   r7: 0x00001003
    r8: 0x00000000   r9: 0x00000000  r10: 0x00000004  r11: 0xa0004308
   r12: 0x900074c0  r13: 0x00000000  r14: 0x00000000  r15: 0x00000001
   r16: 0x00000001  r17: 0x00000000  r18: 0xa0191458  r19: 0x00000000
   r20: 0x0000390f  r21: 0x00000000  r22: 0x00115e48  r23: 0x246b7792
   r24: 0xbfffef80  r25: 0x00000450  r26: 0x00001003  r27: 0x00000000
   r28: 0x00000000  r29: 0x00000000  r30: 0x03000006  r31: 0x90191458

To get the most out of this information, you need a good understanding of the PowerPC runtime architecture. For a detailed description, see Mach-O Runtime Architecture. However, you can still get useful results by applying the following rules of thumb:

• Focus on three values: srr0, lr, and the exception address (described earlier).

• srr0 is the program counter at the time that the exception occurred. That is, it's the address of the instruction that caused the exception.

• lr is typically used to hold the return address of a function call.

• If srr0 is equal to the exception address, the exception was caused by fetching instructions. Typically this means that you've called a bogus function pointer (or, isomorphically, called a method on a bogus object). In this case the return address is typically in lr, which tells you the address of the code that called the bogus function pointer.

• If ssr0 is equal to lr which is equal to the exception address, your program has crashed returning from a function. This typically means that you've corrupted the stack (the return address is typically saved on the stack during the execution of a function) and then died returning to a bogus address.

• If ssr0 is not equal to the exception address, the exception was caused by a memory access instruction (in terms of C, this means that you're dereferencing an invalid pointer).

• Finally, it can be helpful to look through the other registers for telltale signs. For example, if a register contains ASCII characters whose value only appear in one place in your program, that's a clue as to what code has executed recently. Alternatively, if a register contains a well known error value (for example, dskFulErr, which means "disk full error" in the Core Services File Manager, whose value is -34, or 0xffffffde in hex), that might be a clue as to why your program failed.

In the example in Listing 5, you can see that srr0 is 0x00000000, which is equal to the exception address (see Exception Information) but is not equal to lr. Thus, the program has crashed by calling a NULL function pointer and the caller's address is in lr.

Back to Top 

Libraries

The final part of a crash log is a description of all of the libraries loaded into the process. Listing 6 is an example of this.

Listing 6: Libraries

Binary Images Description:
    0x1000 -    0x1bfff com.apple.TextEdit 1.3 (202)    /Applications/TeÖ
0x8fe00000 - 0x8fe4ffff dyld    /usr/lib/dyld
0x90000000 - 0x90122fff libSystem.B.dylib   /usr/lib/libSystem.B.dylib
0x90190000 - 0x9023dfff com.apple.CoreFoundation 6.3.4 (299.31) /System/Ö
0x90280000 - 0x904f9fff com.apple.CoreServices.CarbonCore 10.3.4    /SysÖ
0x90570000 - 0x905defff com.apple.framework.IOKit 1.3.2 (???)   /System/Ö
0x90610000 - 0x9069afff com.apple.CoreServices.OSServices 3.0.1 /System/Ö
0x90700000 - 0x90700fff com.apple.CoreServices 10.3 (???)   /System/LibrÖ
0x90720000 - 0x90787fff com.apple.audio.CoreAudio 2.1.2 /System/Library/Ö
0x907f0000 - 0x907f9fff com.apple.DiskArbitration 2.0.3 /System/Library/Ö
0x90810000 - 0x90810fff com.apple.ApplicationServices 1.0 (???) /System/Ö
[Ö]

This list is particularly useful because you can use it to determine a symbolic backtrace in a program without symbols. It can also be useful if your program makes extensive use of plug-ins because it will show you exactly what plug-ins were loaded in your process. Finally, you can look through this list for libraries that you don't expect to be loaded into your process, such as those used by common application patching (or 'enhancement') technologies.

Back to Top 

Debugging Without Symbols

There are three levels of Mach-O debugging symbols:

• full debugging symbols ó This is what you use during development; it allows you to step through the code in the debugger.

• per-function symbols ó If you don't build with debugger symbols, the linker still includes a symbol for each function in the program.

• only-exported symbols ó If you strip your program, it will only include symbols for functions that are exported (that is, can be imported by other programs or libraries).

A program built with full debugging symbols is huge: you only use this during development. When you ship to end users, you typically choose between per-function symbols and only-exported symbols. Using per-function symbols increases the size of your binary by a few percent but it makes it easier to interpret crash logs. On the other hand, using only-exported symbols makes your binary smaller but your crash logs only contain hex addresses.

The good news is that you can have your cake and eat it too. If you set up your build environment correctly, you can ship a program with export-only symbols to your users and still be able to map addresses in a crash log to their symbolic names. The following sections explain how to do this.

Setting Up Your Build System

The first step is to set up your build system so that it generates a binary with symbols and strips the symbols from that binary to produce a separate binary without symbols. The exact approach to use depends on how your build system is set up. The fundamental mechanism is the strip command line tool. Listing 7 shows how to use this tool to strip symbols from an application. Listing 8 shows how to do the same thing for a shared library or bundle. In this case you have to supply a export file that lists the exported symbols that you want to preserve. If you don't preserve these symbols, programs that import your library (or load you bundle) won't be able to find any exported symbols.

Listing 7: Stripping an application

$ strip -u -r original/MyApp -o stripped/MyApp

Listing 8: Stripping a library

$ strip -u -r -s MyLib.exp original/MyLib -o stripped/MyLib

Back to Top 

Applications

When you receive a crash log for a program whose symbols have been stripped, addresses in the crash log are printed in plain hex. You can use the original, non-stripped program to work out the symbolic name for these addresses. This process is particularly easy for a Mach-O application, whose code is always loaded at the same address (typically 0x0001000). You can simply point the atos command line tool (or GDB) at the original, non-stripped binary. Listing 9 shows an example of how to do this using atos. The -o original/MyApp argument tell atos to operate on the original/MyApp binary. Each of the remaining arguments is an address, in hex, which you want to map to a symbolic name. For each address, atos prints one line of output containing the symbolic name that corresponds to that address.

Listing 9: Address to symbol mapping for an application

$ atos -o original/MyApp 0x00003fbc 0x000045a4 0x00004cb8 0x000035c0
_CopyBundleDevelopmentRegionPretty (BundleInfo.c:335)
_PrintBundleInfo (BundleInfo.c:643)
_main (BundleInfo.c:828)
__start (crt.c:267)

For more information about atos, consult its man page.

Back to Top 

Libraries

The process is slightly more complicated for shared libraries or bundles, whose code can be loaded at an arbitrary address. In this case you have to slide the address based on the load address of the library. The first step is to subtract the address that the library was supposed to be loaded (Aideal) from the address that it was loaded (Areal) to calculate the slide. For most third party libraries (and all bundles), Aideal is zero, in which case the slide is equal to Areal. If your library is prebound to a specific address and it successfully loaded at that address, Aideal equals Areal, and the slide is zero.

Once you have calculated the slide, you can subtract it from the address (X) to determine the slid address (Xslid). This is the address in the library as if the library had not been slid. You can then map that to a symbolic name using atos or GDB.

Listing 10 shows an example of this process. In this case, the crash log shows a crash at address 0x00080f70 (X) and that the library was loaded at address 0x00080000 (Areal). Because the library is a bundle, Aideal is zero. Thus the slide (Areal - Aideal) is 0x00080000. This makes the slid address (Xslid) equal to 0x00000f70 (X - slide). You can then map that address within the non-stripped binary using atos.

Listing 10: Address to symbol mapping for an application

-- crash log excerpts

Thread 0 Crashed:
0   com.apple.carbonbundletemplate   0x00080f70 0x80000 + 0xf70
[Ö]

Binary Images Description:
[Ö]
   0x80000 -    0x80fff com.apple.carbonbundletemplate 1.0  /Volumes/GuyÖ

-- command line

$ atos -o original/MyBundle 0x00000f70
_Nested2 (main.c:3)

-- slide is the difference between Areal and Aideal

slide = Areal - Aideal

-- X is some offset from the start of Areal

X = Areal + offset

offset = X - Areal

-- Xslid is the same offset from the start of the library

Xslid = Aideal + offset
      = Aideal + X - Areal
      = X + (Aideal - Areal)
      = X - (Areal - Aideal)
      = X - slide

-- Xslid is just X minus slide

Back to Top 

CrashReporter Limitations

CrashReporter currently has a number of limitations.

• There is currently no way for third party developers to access the reports submitted via CrashReporter. Apple is aware that there is strong demand for such a facility (r. 3356232). Regardless, your users can still submit crash logs to you manually. Moreover, there's no reason why your application couldn't look at its crash log on each launch and, if it has changed, ask the user whether they want to submit it to your bug tracking system.

• It would be nice if you could configure CrashReporter to automatically launch GDB and let you debug the crashed application. Apple expects to support this feature in a future versions of Mac OS X (r. 2529845).

• CrashReporter does not generate a crash log if your program terminates because of an abort system call (r. 3291139).

• If you write a program that causes an exception but handles that exception via a signal handler, CrashReporter will erroneously generate a crash log for your program (r. 2941263).

Back to Top 

Further Reading

• Technical Q&A QA1288, 'Suppressing the "unexpectedly quit" alert'

• Mach-O Runtime Architecture

• Debugging with GDB

• atos man page

• strip man page

Back to Top 

Document Revision History

Posted: 2004-09-09